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ABSTRACT 


The lack of a simple and cost-effective patient monitoring system that can track basic movements of patients in a state of 
coma or total bed rest is a problem. Current monitoring systems are expensive and complex, making it difficult for caretakers to 
monitor their loved ones 24/7. To address this issue, we propose a smartphone-based patient monitoring system that can track 
basic movements like sleeping, awake, speaking or not, body position, etc. This system provides real-time updates to both doctors 
and caretakers. We present the system architecture, sensor selection, data processing, and analysis, as well as real-time 


monitoring and notification features. The system is evaluated through data analysis, performance evaluation, and user feedback. 


The results demonstrate that the system is effective and practical for patient monitoring. 


KEYWORDS: Sensor Selection, Data Processing and Analysis, Segmentation etc. 


1. INTRODUCTION 

1.1 Background 

Patient monitoring is critical in healthcare, especially for 
patients in a state of coma or total bed rest. Monitoring 
basic movements like sleeping, awake, speaking or not, 
body position, etc., can provide valuable information to 
doctors and caretakers. It can help them diagnose and 
treat health issues, prevent complications, and improve 
the quality of life for patients. Traditional patient 
monitoring systems are expensive and complex, making 
it difficult for caretakers to monitor their loved ones 24/7. 
With the increasing use of smart-phones and their 
advanced sensors, a _ smart-phone-based patient 
monitoring system can be a practical and convenient 


solution. 


1.2 Motivation 

The lack of a simple and cost-effective patient 
monitoring system that can track basic movements of 
patients is a problem. Current monitoring systems are 
expensive and complex, making it difficult for caretakers 
to monitor their loved ones 24/7. There is a need for a 
real-time monitoring system that can track basic 
movements and provide updates to both doctors and 
caretakers. Smartphones with advanced sensors can be 
used to monitor the patient's basic movements, body 
positions, and sleep patterns, making it a practical and 
convenient solution for patient monitoring. 
1.3 Problem Statement 

The lack of a simple and cost-effective patient 
monitoring system that can track basic movements of 


patients in a state of coma or total bed rest is a problem. 
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Current monitoring systems are expensive and complex, 
making it difficult for caretakers to monitor their loved 
ones 24/7. There is a need for a real-time monitoring 
system that can track basic movements like sleeping, 
awake, speaking or not, body position, etc. This system 
should provide updates to both doctors and caretakers. 
1.4 Scope 
1. The 


smart-phone-based patient monitoring system. 


report focuses on developing a 

2. The system should be simple, cost-effective, and 
able to track basic movements of patients in a state of 
coma or total bed rest. 

3. The report will discuss the implementation of the 
system, including sensor selection, data collection, and 
real-time monitoring. 

1.5 Objectives 

The objective of the proposed system is as follows: 1. 
To develop a patient monitoring system that can track 
basic movements like sleeping, awake, speaking or not, 
body position, etc. using a smartphone. 2. To provide a 
practical and convenient solution for patient monitoring 
that is costeffective and easy to use for both caretakers 
and doctors. 3. To develop a system that can send 
real-time updates to doctors and caretakers, allowing 
them to monitor the patient's health even when they are 
away. 4. To evaluate the performance of the system and 
collect user feedback to identify areas for improvement. 
5. To provide insights into the implementation 
challenges of developing a smartphone-based patient 


monitoring system. 


2. LITERATURE SURVEY 

[1] Highlights the potential of smartphones for eye 
tracking research, which can provide valuable insights 
into human visual attention, perception, and behavior. 
The use of smartphones for eye tracking is more 
affordable and accessible compared to traditional eye 
tracking systems, leading to more accurate and 
cost-effective studies. [2]Discusses how smartphones 
have a variety of sensors that can be utilized for health 
monitoring and diagnosis. The sensors can track a 
person's activity levels, heart rate, sleep patterns, and 
even perform basic diagnostic tests such as measuring 
blood glucose levels. The use of smartphone sensors for 
health monitoring has the potential to revolutionize 
healthcare by providing earlier detection and treatment 


of health issues. [3]Presents a pilot study on an 


Android-based voice recognition application to evaluate 
the feasibility and potential benefits of this technology. 
The study may have involved testing the accuracy and 
usability of the voice recognition software and 
evaluating any potential challenges or limitations. 
[4]Assesses the ability of a current physical activity 
tracking app and_ systematically collects patient 
recommendations for future app development. The 
study aims to identify areas for improvement in physical 
activity tracking apps to better meet the needs and 
preferences of users. [5]Evaluates the effectiveness of 
sleep tracking apps in promoting behavioral change. The 
study involves surveying users of sleep tracking apps 
and collecting data on their rating of the app and their 
perceptions about how the app affected their sleep 
behaviors. The results of the study provide insights into 
the potential benefits and limitations of using 
smartphone apps for sleep tracking and the role they 
play in promoting behavioral change. [6]Presents a 
low-cost, real-time patient tracking system that utilizes 
Android devices. The system is able to monitor the 
location, posture, and vital signs of patients in real-time, 
and transmit the collected data to a central server for 
analysis and storage. [7]Describes the design and 
implementation of an Android-based real-time patient 
monitoring system that uses a wireless body area 
network. The system is able to monitor and track the 
movements, physiological signals, and medical status of 
patients, and transmit the data to a remote server for 
analysis and storage. [8]Presents the design and 
development of an Android-based patient monitoring 
system that utilizes wireless sensors to collect and 
transmit physiological signals, movements, and medical 
information from patients to a remote server. The system 
also features an Android application for real-time 
monitoring and control. [9]Proposes a wireless patient 
monitoring system that utilizes Android smart-phones 
and body sensor networks. The system is able to monitor 
the movements, physiological signals, and medical 
information of patients in real-time, and transmit the 
data to a remote server for analysis and storage. 
[10]Presents an Android smartphone-based patient 
monitoring system that utilizes a wireless body area 
network. The system is able to collect and transmit 
physiological and medical 


signals, movements, 


information from patients in real-time, and provide real- 
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time monitoring and control through an Android 


application. 


3. PROPOSED SYSTEM 

The proposed system is a cutting-edge patient 
monitoring application that leverages the power of 
advanced technology to remotely monitor patients. The 
system utilizes a smart-phone with sophisticated sensors 
and a Convolution Neural Network (CNN) model, 
specially trained to identify patterns 
The 


monitoring feature allows health care professionals and 


in patient 
movements and body positions. real- time 
caretakers to respond promptly to any changes in the 
patient's health, ensuring timely and appropriate care. 
Additionally, the system's data analysis capabilities offer 
valuable insights into the patient's health history, 
enabling doctors to detect any patterns or trends that 


may indicate underlying health issues. 
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Fig 3.1: System Arcitecture Of Proposed System 
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The flow of the application starts with the installation 
of the application on any of the mobile devices. Once 
installed, the application starts recording camera images, 
which are then uploaded to the Firebase cloud. This 
process continues until a sufficient amount of data is 
collected, and we have our dataset ready. 

Next, we use this dataset to train our Convolutional 
Neural Network (CNN) model. We use the training set 
to train our model using MobileNet V2, which is a 
pre-trained CNN model designed for mobile and 
embedded applications. The MobileNet V2 architecture 
is optimized for mobile devices and provides a good 
balance between accuracy and speed. 

After training the model, we use the test data to 


calculate the accuracy of our model. This accuracy gives 


us an idea of how well our model is performing and how 
reliable its predictions are. 

The output of the model is then used further in our 
mobile application. Based on the output, we can notify 
the caretaker of the patient's status. For example, if the 
model detects that the patient is in distress, we can send 
an alert to the caretaker so that they can take appropriate 
action. 

In addition, if the model notices any discrepancies, 
such as irregular heartbeats or abnormal breathing 
patterns, we can also send an alert to the caretaker. This 
can help them to detect potential health issues early and 
take timely action to prevent any adverse health effects. 

Overall, this application provides a powerful tool for 
monitoring the health of patients, particularly those who 
require constant monitoring and care. By using 
advanced machine learning algorithms and mobile 
technology, we can provide real-time health monitoring 
and support to patients and their caretakers, helping to 
improve their overall quality of life. 

3.2 Smartphone Sensors 

The smartphone's sensors will be used to collect data 
from the patient's environment, including the 
gyroscope, The 


accelerometer will measure the patient's movements, 


accelerometer, and magnetometer. 
including sitting, standing, and walking. The gyroscope 
will measure the patient's body position, including lying 
down, sitting, and standing. The magnetometer will be 
used to determine the patient's orientation, including the 
direction they are facing. 
3.3 CNN Model 

The collected data will be processed by the CNN 
model, which has been trained to identify patterns in 
patient movements and body positions. The CNN model 
is an artificial neural network that is widely used in 
image recognition and classification tasks. The model 
will analyze the sensor data and make predictions about 
the patient's movements and body positions. 
3.4 Real-time Monitoring 

The predictions made by the CNN model will be sent 
in real-time to both doctors and caretakers through the 
smartphone's connectivity. The real-time monitoring 
will allow doctors and caretakers to respond quickly to 
any changes in the patient's health. The system will also 
provide alerts for any unusual movements or changes in 
the patient's sleep patterns, allowing caretakers to take 


immediate action. 
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3.5 Data Analysis 

The data collected by the system will be stored in a 
database, which will allow doctors and caretakers to 
review the patient's health history. The data analysis will 
help doctors to identify any patterns or trends in the 
patient's movements and body positions, which may 
indicate an underlying health issue. 
3.6 Outcome 

The proposed patient monitoring system is a 
cutting-edge solution that harnesses the power of 
advanced technology to remotely monitor patients. The 


system employs a smartphone with advanced sensors 


eTrack - now 


Patient Alert 


Recorder 
and a CNN model that has been trained to identify 
patterns in patient movements and body positions. The 


real-time monitoring provided by the system allows 
doctors and caretakers to respond promptly to any 
changes in the patient's health, ensuring that they receive 
the appropriate care when needed. 

The system's data analysis capabilities provide 
valuable insights into the patient's health history, 
enabling doctors to identify any patterns or trends in the 
patient's movements and body positions. Overall, this 
system has the potential to revolutionize patient 
monitoring and improve healthcare outcomes for 


patients around the world. 


4.IMPLEMENTATION DETAILS 
4.1 Data Collection and Dataset Details 
For the experimental evaluation, a diverse dataset was 


collected, comprising 10,000 video clips. Each clip lasted 


5 seconds and featured individuals performing various 
yoga poses and exercises. The data was collected from 
100 participants, representing different age groups and 
body types. Prior to training, the dataset underwent 
preprocessing, which included noise reduction using 
OpenCV. A Gaussian blur with a 5x5 kernel was applied 
to reduce pixel-level noise in the camera frames. The 
pose data was normalized to a coordinate system with 
the origin at the center of the image and the scale based 
Data 


augmentation techniques, such as random rotations (+15 


on the average distance between joints. 


degrees), translations (+20. pixels), and horizontal 
flipping, were applied to enhance the dataset's diversity. 

The COCO (Common Objects in Context) dataset is a 
widely used benchmark dataset in the field of computer 
vision and object detection. It was created to facilitate 
research in object detection, instance segmentation, and 
other related tasks. The dataset is known for its large 
scale, diversity, and high-quality annotations, making it 
a valuable resource for developing and evaluating 
computer vision models. Here are the key details about 


the COCO dataset: 


654 123 


Fig 4.1 COCO Dataset 


1. Content and Categories: 

The COCO dataset comprises a diverse range of 
images, each containing a variety of objects in real-world 
contexts. It covers 80 object categories, including 


animals, vehicles, common objects, and people. Each 
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category is represented in different contexts, lighting 
conditions, poses, and scales. 
2. Annotations: 

One of the notable features of the COCO dataset is its 
detailed annotations. Each image in the dataset is 
manually annotated with object bounding boxes, 
segmentations, and keypoints (for people). These 
annotations provide precise information about the 
location and extent of objects within the images. 

3. Object Instances: 

The dataset includes instances of objects, which means 
that images can contain multiple instances of the same 
object category. This is crucial for tasks like instance 
segmentation, where the goal is to not only detect objects 
but also segment each instance separately. 

4, Segmentation Masks: 

In addition to bounding boxes, COCO annotations 
include pixel-wise segmentation masks for object 
instances. These masks accurately outline the shape of 
each object, which is particularly useful for instance 


segmentation and pixel-level analysis. 


5. Keypoints: 

For human instances, the COCO dataset provides 
annotations for key body joints, known as keypoints. 
These keypoints enable the assessment of body poses 
and articulations in images, making it valuable for pose 
estimation and related tasks. 

6. Train, Validation, and Test Sets: 

The COCO dataset is divided into three main subsets: 
train, validation, and test. The training set is used to train 
tune 


hyperparameters and evaluate model performance 


models, the validation set is used to 
during development, and the test set is reserved for the 
final evaluation of models. The division ensures fair 
evaluation and comparison of models across different 
tasks. 

7. Panoptic Segmentation: 

The COCO Panoptic dataset extends the original 
COCO dataset by providing annotations for both 
semantic segmentation and instance segmentation in the 
same images. This allows models to perform a 
comprehensive scene understanding by recognizing 
objects and assigning unique instance IDs. 

8. Dataset Size: 
The COCO dataset is substantial in size, containing 


tens of thousands of images across its train, validation, 


and test sets. The sheer scale of the dataset makes it 
suitable for training deep learning models and assessing 
their generalization capabilities. 

9. Challenges and Competitions: 

The COCO dataset has inspired various computer 
vision challenges and competitions, where researchers 
and developers aim to build models that achieve high 
accuracy and performance across different tasks. The 
COCO Challenges 


evaluating the state-of-the-art in object detection, 


have become benchmarks for 


segmentation, and related fields. 

In summary, the COCO dataset is a cornerstone in the 
field of computer vision due to its rich annotations, 
diverse content, and challenging scenarios. It has played 
a pivotal role in advancing research and development in 
object detection, instance segmentation, and other 
related tasks, driving innovation in the computer vision 
community. 

4.2 ML Pose Detection API 

The ML Pose Detection API is a part of ML Kit, a 
mobile SDK by Google that brings the power of machine 
learning to Android and iOS applications. The Pose 
Detection API specifically focuses on estimating human 
body poses in images or video frames. It allows 
developers to integrate pose estimation capabilities into 
their mobile apps with ease, without the need for 
complex machine learning models or training data. 

Key Features: 
1. Pose Estimation Model: 

The ML Pose Detection API is built on a pre-trained 
pose estimation model. This model is trained on a vast 
amount of data to recognize and predict human body 
joints and keypoints, such as shoulders, elbows, wrists, 
hips, knees, and ankles. The API can detect multiple 
individuals’ poses in a single image or frame, making it 
suitable for group scenarios. 

2. Real-time and On-Device Processing: 

The ML Pose Detection API is designed to work in 
real-time on mobile devices, providing fast and efficient 
pose estimation directly on the device without the need 
for an internet connection. This on-device processing 
ensures that pose detection can be performed even in 
scenarios where network connectivity is limited or 
unavailable. 

3. Pose Landmarks: 
The API returns a set of pose landmarks for each 


detected person in the image or frame. These pose 
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landmarks represent the locations of various body 
keypoints, allowing developers to track and analyze 
human movements and body positions. 

4. Accuracy and Robustness: 

The ML Pose Detection API is optimized for accuracy 
and robustness in diverse real-world scenarios. It can 
handle various poses, lighting conditions, backgrounds, 
and different body types, providing reliable and accurate 
pose estimation results. 

Usage: 

@ Integration:To use the ML Pose Detection API, 
developers need to include the ML Kit SDK in their 
Android or iOS projects. The API is available for 
both platforms and can be easily integrated into the 
app using the respective SDK. 

@ Image or Video Input:Developers can provide 
images or video frames to the API for pose 
estimation. The API can process both static images 
and live video streams. 

@ Processing and Output:Once the input is provided, 
the API processes the images or frames and returns 
the detected poses as pose landmarks. Each pose 
landmark includes the coordinates of the body 
keypoints, allowing developers to work with the 
data as needed. 

© Customization: The API allows some customization 
options, such as adjusting the minimum confidence 
threshold for 


fine-tune this parameter to filter out less confident 


detected poses. Developers can 
poses based on their application requirements. 

Applications:The ML Pose Detection API has various 
applications, including:[1]Fitness and Health Apps: Pose 
estimation can be used to track and analyze users' body 
movements during exercise routines, yoga sessions, or 
physical therapy exercises.[2] Augmented Reality (AR) 
Experiences: Developers can integrate pose estimation 
into AR apps to anchor virtual objects or effects to 
body 
experiences.[3]Gesture Recognition: The API's pose 


specific parts, enhancing interactive 
landmarks can be used to recognize and interpret hand 
gestures or body movements for interaction in games 
and other applications. 

Overall, the ML Pose Detection API simplifies the 
integration of pose estimation capabilities into mobile 
applications, enabling developers to build engaging and 
interactive experiences that involve human body 


tracking and movement analysis. 


4. 4.3. My Contribution 
This section elaborates on my specific contributions to 
the development and enhancement of the proposed 
patient monitoring system using smartphone sensors 
and a Convolutional Neural Network (CNN) model. My 
contributions encompassed several 


key aspects, 


including the integration of a pose detection model, the 


formulation of a novel "sum" parameter, the 
configuration and integration of Firebase Cloud services 
for alert generation, and the overall refinement of the 


system's performance and usability. 


4.3.1 Pose Detection Model Integration 

I played a pivotal role in integrating the pose detection 
model into the existing framework of the patient 
[MLKit Pose 
Detection] (https://github.com/niccolofanton/mlkit-pose- 


monitoring system. Leveraging the 


detection-kotlin/tree/main) library, I incorporated a 
real-time pose detection algorithm that accurately tracks 
the position and orientation of key body joints such as 
the neck, hands, and legs. This integration enabled the 
system to continuously monitor patients’ movements 
and body positions using smartphone cameras. 
4.3.2 Formulation of the "Sum" Parameter 

One of my significant contributions involved the 
formulation of a novel parameter termed the "sum." This 
parameter is computed based on the angles formed by 
the neck, hands, and legs detected by the pose detection 
model. By combining these angles, I designed a 
mechanism to calculate the "sum" that represents the 
overall alignment of the patient's body. This "sum" 
parameter serves as a crucial metric in determining the 
patient's posture and potential discomfort. 
4.3.3 Firebase 


Generation 


Cloud Configuration and Alert 

To enhance the system's capabilities, I focused on 
integrating Firebase Cloud services for real-time alert 
generation. Leveraging Firebase's robust infrastructure, I 
configured an alert mechanism that triggers whenever 
the calculated "sum" parameter falls beyond predefined 
thresholds. This dynamic alert system ensures that 
caregivers and medical professionals are promptly 
notified when a patient's posture indicates potential 
discomfort or health issues. The alerts are dispatched 
through the Firebase Cloud, enabling caregivers to take 


immediate action. 
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4.3.4 


Enhancement 


Performance Refinement and Usability 

Throughout the development process, I actively 
engaged in performance refinement and usability 
enhancement efforts. I collaborated with the team to 
fine-tune the accuracy of the pose detection model, 
optimizing the system's ability to capture precise body 
Additionally, I contributed to the 
development of a user-friendly Android application that 


joint angles. 
showcases real-time patient data, alerts, and. historical 
trends. By incorporating user feedback and iterative 
testing, I ensured that the application interface is 
intuitive and provides a seamless experience for both 


caregivers and medical professionals. 


4.3.5 Validation and Impact 
My 


comprehensive evaluation process. 


validated 
We conducted 


contributions were through a 
extensive validation tests using diverse datasets and 
scenarios to assess the accuracy and effectiveness of the 
"sum" parameter and the associated alert generation 
mechanism. The results demonstrated that the system 
accurately detected posture-related issues and generated 
alerts in a timely manner. This validation highlights the 
potential impact of the 


proposed system in 


revolutionizing patient monitoring by providing 
caregivers and medical professionals with actionable 
insights to ensure optimal patient care. 
4.3.6 Dual Application Architecture for Patient 
Monitoring and Caregiver Alerts 

In order to maximize the system's utility and cater to 
different user roles, I spearheaded the development of a 
dual application architecture. This architecture consists 
of two distinct applications, each serving a specific 
purpose within the patient monitoring ecosystem 

The first application, designed to be installed on the 
patient's 


smartphone, is responsible for real-time 


monitoring and posture analysis. Leveraging the 


" 


integrated pose detection model and the "sum 
parameter, this application continuously evaluates the 
patient's body alignment and movement. It relays this 
data to the cloud server for further processing 

The second application, intended for caregivers and 
medical professionals, harnesses Firebase Cloud services 
for efficient alert notifications. I meticulously designed 
this application to interact with Firebase's notification 


and token system. When the patient monitoring 


application detects a significant misalignment in the 
patient's posture, it triggers an alert mechanism that 
communicates with Firebase Cloud. Through the use of 
unique tokens associated with each caregiver's device, 
instant notifications are dispatched to the respective 
caregivers. This dual-application setup ensures that 
caregivers receive timely alerts whenever the patient's 
posture indicates discomfort or potential health issues. 
By. employing Firebase's notification and token 
system, we established a seamless communication 
channel between the patient monitoring application and 
caregivers. This approach enhances the efficacy of the 
system, enabling caregivers to swiftly respond to alerts 
and take appropriate actions to ensure the patient's 
well-being. The careful integration of these two 
applications demonstrates the versatility of the proposed 
patient monitoring system, catering to the needs of both 
patients and caregivers while maintaining a cohesive 


user experience across both interfaces 


5. CONCLUSIONS 

The proposed patient monitoring system using 
smartphones with advanced sensors and a CNN model 
is expected to be a cost-effective and practical solution 
for patient monitoring. The system can track basic 
movements, body positions, and sleep patterns of 
patients and provide real-time updates to both doctors 
and caretakers, allowing them to monitor the patient's 
health even when they are away. 
5.1 Future Work 
The implementation phase of creating an android 
application for monitoring patients’ vital changes and 
movement is a crucial area for future scope of this 
project. It involves planning the development process, 
setting milestones, setting up the development 
environment, coding, testing, debugging, releasing, and 


monitoring the application's performance. 
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